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Investor Sentiment in the Stock Market 


Malcolm Baker and Jeffrey Wurgler 


he history of the stock market is full of events striking enough to earn their 

own names: the Great Crash of 1929, the "Tronics Boom of the early 1960s, 

the Go-Go Years of the late 1960s, the Nifty Fifty bubble of the early 1970s, 
the Black Monday crash of October 1987, and the Internet or Dot.com bubble of 
the 1990s. Each of these events refers to a dramatic level or change in stock prices 
that seems to defy explanation. The standard finance model, in which unemotional 
investors always force capital market prices to equal the rational present value of 
expected future cash flows, has considerable difficulty fitting these patterns. Re- 
searchers in behavioral finance have therefore been working to augment the 
standard model with an alternative model built on two basic assumptions. 

The first assumption, laid out in Delong, Shleifer, Summers, and Waldmann 
(1990), is that investors are subject to sentiment. Investor sentiment, defined 
broadly, is a belief about future cash flows and investment risks that is not justified 
by the facts at hand. The second assumption, emphasized by Shleifer and Vishny 
(1997), is that betting against sentimental investors is costly and risky. As a result, 
rational investors, or arbitrageurs as they are often called, are not as aggressive in 
forcing prices to fundamentals as the standard model would suggest. In the 
language of modern behavioral finance, there are limits to arbitrage. Recent stock 
market history has cooperated nicely, providing the Internet bubble and the 
ensuing Nasdaq and telecom crashes, and thus validating the two premises of 
behavioral finance. A period of extraordinary investor sentiment pushed the prices 
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of speculative and difficult-to-value technology stocks to unfathomable levels in the 
late 1990s. Instead of creating opportunity for contrarian arbitrageurs, the period 
forced many such arbitrageurs out of business, as prices that were merely high went 
higher still before an eventual crash. 

Now, the question is no longer, as it was a few decades ago, whether investor 
sentiment affects stock prices, but rather how to measure investor sentiment and 
quantify its effects. One approach is “bottom up,” using biases in individual investor 
psychology, such as overconfidence, representativeness, and conservatism, to ex- 
plain how individual investors underreact or overreact to past returns or funda- 
mentals. A related class of models, discussed by Hong and Stein in this issue, or 
Shefrin (2005), relies on differences of opinion across investors, sometimes com- 
bined with short sales constraints, to generate misvaluation. When aggregated, 
these models make predictions about patterns in marketwide investor sentiment, 
stock prices, and volume. 

The investor sentiment approach that we develop in this paper is, by contrast, 
distinctly “top down” and macroeconomic. The starting point for this approach is 
that many of the bottom-up models lead to a similar reduced form of variation over 
time in mass psychology; and it is certain that none of the models is uniquely true. 
Real investors and markets are too complicated to be neatly summarized by a few 
selected biases and trading frictions. The top-down approach focuses on the 
measurement of reduced-form, aggregate sentiment and traces its effects to market 
returns and individual stocks. The new directions in this top-down approach build 
on the two broader and more irrefutable assumptions of behavioral finance— 
sentiment and limits to arbitrage—to explain which stocks are likely to be most 
affected by sentiment, rather than simply pointing out that the level of stock prices 
in the aggregate depends on sentiment.” 

In particular, stocks of low capitalization, younger, unprofitable, high-volatility, 
non—dividend paying, growth companies or stocks of firms in financial distress are 
likely to be disproportionately sensitive to broad waves of investor sentiment. As the 
reader will recall, small startup firms represented a majority of the excitement and 
subsequent carnage of the Internet bubble, so this statement may ring true already. 
Theoretically, it follows because 1) these categories of stocks tend to be harder to 
arbitrage (for example, they have higher transaction costs) and 2) they are more 
difficult to value, making biases more insidious and valuation mistakes more likely. 

The remainder of the paper develops these theoretical predictions in more 
detail, shows how one might measure investor sentiment explicitly, and finally 
explains how to use the sentiment measures to validate the key predictions of the 
top-down approach. Certainly, both the bottom-up and top-down approaches to 
investor sentiment deserve continued attention. The advantage of the top-down 


' See Barberis, Shleifer, and Vishny (1998) and Daniel, Hirshleifer, and Subrahmanyman (1998) for 
models of this sort. 

? As an analogy, aggregate risk aversion is another one-dimensional variable that will affect all stocks to 
some degree but will also affect some more than others. 
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approach is its potential to encompass bubbles, crashes, and more everyday pat- 
terns in stock prices in a simple, intuitive, and comprehensive way. The advantage 
of the bottom-up model is in providing microfoundations for the variation in 
investor sentiment that the top-down model takes as exogenous. 


Theoretical Effects of Investor Sentiment on Stocks 


A pioneering and well-known set of studies of sentiment and aggregate stock 
returns appeared in the 1980s. They were largely atheoretical, testing in various 
ways whether the stock market as a whole could be mispriced. Authors looked for: 
the tendency of aggregate returns to mean revert; volatility in aggregate stock index 
returns that could not be justified by volatility in fundamentals, which is in fact 
another way of characterizing mean reversion in returns; or predictability of 
aggregate returns using simple valuation ratios like the ratio of aggregate dividends 
to stock market value.” 

In these studies, the role of sentiment was left implicit, and the statistical 
evidence was not usually very strong. Practically speaking, it is hard to distinguish 
a random walk from a long-lived bubble, especially in a short time series containing 
at best a few bubbles. Even when statistical inferences seemed robust, the economic 
interpretation was still unclear. Predictability of stock returns could reflect the 
correction of sentiment-induced mispricings or, arguably, time-varying risk or risk 
aversion that causes time variation in expected stock returns. 

More recent studies, such as Baker and Wurgler (2006), utilize interim ad- 
vances in behavioral finance theory to provide sharper tests for the effects of 
sentiment. In particular, in the many behavioral models of securities markets 
inspired by DeLong, Shleifer, Summers, and Waldmann (1990), investors are of 
two types: rational arbitrageurs who are sentiment-free and irrational traders prone 
to exogenous sentiment. They compete in the market and set prices and expected 
returns. But rational arbitrageurs are limited in various ways. These limits come 
from short time horizons or from costs and risks of trading and short selling. As a 
result, prices are not always at their fundamental values. In such models, mispricing 
arises out of the combination of two factors: a change in sentiment on the part of 
the irrational traders, and a limit to arbitrage from the rational ones. 

The key predictions of this framework come from its two moving parts. 
Consider first the possibility that sentiment-based demand shocks vary across firms, 
while arbitrage is equally difficult across firms. For example, suppose one thinks 
about investor sentiment as the propensity to speculate by the marginal investor, 
akin to a propensity to play the lottery; then sentiment almost by definition is a 
higher demand for more speculative securities. So when sentiment increases, we 
expect such “speculative” stocks to have contemporaneously higher returns. 


° See Shiller (1981) on excess volatility; Fama and French (1988) and Poterba and Summers (1988) on 
mean reversion; and Campbell and Shiller (1988) and Fama and French (1989) on valuation ratios. 
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What makes some stocks more speculative than others? We believe that the 
crucial characteristic is the difficulty and subjectivity of determining their true 
values. For instance, in the case of a young, currently unprofitable, but potentially 
extremely profitable growth firm, the combination of no earnings history and a 
highly uncertain future allows investors to defend valuations ranging from much 
too low to much too high, as befits their prevailing sentiment. During a bubble, 
when the propensity to speculate is high, investment bankers can join the chorus 
arguing for high valuations. By contrast, the value of a firm with a long earnings 
history, tangible assets, and stable dividends is much less subjective, and thus its 
stock is likely to be less sensitive to sentiment. One could appeal to psychological 
foundations here. Uncertainty means that the effect of overconfidence (Daniel, 
Hirshleifer, and Subrahmanyman, 1998), representativeness, and conservatism 
(Barberis, Shleifer, and Vishny, 1998) is more pronounced. Further, differences of 
opinion (Miller, 1977), even when investors have the same basic information, can 
be large. The changes over time in these biases are what we would call shifts in the 
propensity to speculate. 

Now suppose instead that we view investor sentiment as simply optimism or 
pessimism about stocks in general, and we allow the limits to arbitrage to vary across 
stocks. A large body of research shows that arbitrage tends to be particularly risky 
and costly for certain stocks: namely those that are young, small, unprofitable, or 
experiencing extreme growth. Such stocks tend to be more costly to buy and to sell 
short (D’Avolio, 2002). Such stocks have a high degree of idiosyncratic variation in 
their returns, which makes betting on them riskier (Wurgler and Zhuravskaya, 
2002). Such stocks’ higher volatility may lead to second-guessing by the investors 
who provide funds to the arbitrageur, ultimately leading to withdrawals from 
contrarian arbitrageurs just when the mispricing is greatest (Shleifer and Vishny, 
1997). By not paying dividends, such stocks’ fundamentals remain far in the future 
and therefore subject to speculation (Pontiff, 1996). Thus, again, we might expect 
that sentiment has a greater effect on such stocks. 

The key point is that in practice, the same securities that are difficult to value also tend 
to be difficult to arbitrage. Therefore, we are left with a very robust and testable 
conclusion: The stocks most sensitive to investor sentiment will be those of com- 
panies that are younger, smaller, more volatile, unprofitable, non—dividend paying, 
distressed, or with extreme growth potential (or companies having analogous 
characteristics). Conversely, “bond-like” stocks will be less driven by sentiment. 
Again, note that this assessment does not depend on specifying a fine definition of 
investor sentiment or rely on just one arbitrage mechanism such as short-sales 
constraints. 


The Sentiment Seesaw 

Figure 1 summarizes this perspective into a simple, unified view of the effects 
of sentiment on stocks. The x-axis orders stocks according to how difficult they are 
to value and arbitrage. Bond-like stocks, such as regulated utilities, are toward the 
left; stocks of companies that are newer, smaller, more volatile, distressed, or 
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Figure 1 
Theoretical Effects of Investor Sentiment on Different Types of Stocks 
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Note: Stocks that are speculative and difficult to value and arbitrage will have higher relative valuations 
when sentiment is high. 


extreme growth are toward the right. The y-axis measures prices, with P* denoting 
fundamental values, which, of course, can vary over time. The lines then illustrate 
the basic hypotheses about how stock valuations are affected by swings in sentiment. 
High sentiment should be associated with high stock valuations, particularly for the 
stocks that are hardest to value and to arbitrage. Low sentiment works in the reverse 
direction. In the absence of sentiment, stocks are, on average, assumed to be 
correctly priced at P*. 

An empirical question that arises in the drawing of Figure | is where to locate 
the crossing point of this seesaw. One case (not in Figure 1) is that no crossing 
point exists: the upward-sloping high-sentiment line lies entirely above the no- 
sentiment P* line, which in turn lies entirely above the downward-sloping low- 
sentiment line. That is, when sentiment increases, all stocks’ prices go up, but some 
more than others. In this case, the aggregate effects of sentiment will be strong, 
because aggregate stock indexes are simply averages of the underlying stocks. 

As drawn, Figure | reflects the more complex case where the prices of 
particularly safe, easy-to-arbitrage stocks actually are inversely related to sentiment. 
This outcome could occur if sentiment fluctuations induce substantial changes in 
the demand for speculative securities, for example engendering “flights to quality” 
within the stock market. Such episodes may, controlling for any changes in funda- 
mentals, reduce the prices of speculative stocks and at the same time increase the 
prices of bond-like stocks. In this case, the effect of sentiment on aggregate returns 
will be muted because stocks are not all moving in the same direction. 
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Behavioral theory thus delivers clear cross-sectional predictions about the 
effects of sentiment—but the aggregate predictions are somewhat less clear, which 
may help to explain why the 1980s studies did not always reach strong statistical 
conclusions. The rest of the paper reviews some empirical evidence regarding three 
critical aspects of Figure 1. 

First, we discuss how investor sentiment can be empirically measured. 

Second, we ask whether more speculative and harder-to-arbitrage stocks are 
indeed more sensitive to sentiment, in the sense that their prices co-move more 
with an index of sentiment changes. In finance parlance, we ask whether specula- 
tive and harder-to-arbitrage stocks have higher “sentiment betas.” (The term is 
analogous to the famous concept of “market beta,” which measures the exposure of 
a stock to returns on the stock market as a whole. A stock with a market beta of 1.0 
appreciates by 1 percentage point, on average, when the market return is one 
percentage point. Market betas above 1.0 indicate relatively high market risk 
exposure.) We also test whether bond-like stocks have negative sentiment betas, 
that is, their returns are negatively related to changes in sentiment, as represented 
in Figure 1. 

Third, we investigate whether current investor sentiment levels predict future 
returns as sentiment wanes (perhaps spurred by fundamental news or an absence 
thereof) or as arbitrage forces eventually accumulate to correct mispricings. This 
test is important, because sentiment measures may, despite our best efforts, be 
contaminated by economic fundamentals, and fundamentals should of course 
affect stock returns contemporaneously. In other words, the co-movement patterns 
are subject to a correlated omitted variables critique. Return predictability helps to 
address this concern because it suggests a profitable trading strategy, which by 
definition cannot exist if stocks are priced correctly. 


Ruling out Other Explanations 

Contrasting Figure | with some other finance frameworks helps to clarify the 
unique predictions of the sentiment model. The classical risk-based and the behav- 
ioral disagreement models make distinct predictions about the slope of the overall 
valuation line (the solid line on Figure 1). Neither makes predictions about 
valuations conditional on sentiment (the dashed or dotted lines). 

In the risk-based asset pricing models, such as the capital-asset pricing model, 
a stock’s expected return depends on its risk exposure, measured by market beta, 
times the market risk premium, which is the expected return on the stock market 
as a whole. Furthermore, since investors are rational and risk averse in these 
models, the market risk premium is always positive, though it may change over 
time. See Fama and French (2004) and Perold (2004) for introductions to this 
model in this journal. 

What do these models imply for tests of return predictability? Even if specu- 
lative and hard-to-arbitrage securities have higher market betas, as is likely the case, 
classical models predict that such stocks always have higher expected returns than 
bond-like stocks. In fact, as we will see below, this is not true. When sentiment is 
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measured to be high, speculative and hard-to-arbitrage stocks have lower future 
returns on average than bond-like stocks. This finding is a powerful confirmation 
of the sentiment-driven mispricing view. 

In a behavioral model of disagreement among regular investors combined with 
short-sales constraints by arbitrageurs, on the other hand, hard-to-short stocks can 
become overvalued. As fundamentals are revealed, this mispricing will disappear, so 
the future returns of such stocks will be relatively low on average. The sentiment 
seesaw in Figure 1 makes a distinct and testable prediction that hard-to-arbitrage 
stocks can, conditional on the state of sentiment, be undervalued as well. We will 
also investigate this point. 


Measuring Investor Sentiment 


Investor sentiment is not straightforward to measure, but there is no funda- 
mental reason why one cannot find imperfect proxies that remain useful over time. 
We discuss some generic issues involved in measuring sentiment and describe 
proxies for sentiment that have come into use. We then describe a sentiment index 
that combines several of these proxies, and we show that it fluctuates with the major 
speculative episodes of the past 40 years. 


Potential Sentiment Proxies 

An exogenous shock in investor sentiment can lead to a chain of events, and 
the shock itself could in principle be observed at any or every part of this chain. For 
example, it might show up first in investor beliefs, which could be surveyed. These 
beliefs might then translate to observable patterns of securities trades, which are 
recorded. Limited arbitrage implies that these demand pressures might cause some 
mispricings, which might be observed using benchmarks for fundamental value like 
the book-to-market ratio. These mispricings might engender an informed response 
by insiders, such as corporate executives, who may have both the superior infor- 
mation and the incentive to take advantage of it, and the patterns of firms choosing 
to adjust their balance of equity or debt could be observed. 

The bad news is that each part of this chain is also subject to confounding 
influences. Economists always treat surveys with some degree of suspicion, because 
of the potential gap between how people respond to a survey and how they actually 
behave. Trades net to zero, so measuring sentiment with trading activity means 
taking a stand on the identity of irrational investors. Market prices of securities 
normally reflect fundamentals, by and large, with sentiment playing a lesser role. 
Corporations may alter their financial structure for many reasons, including 
a change in business fundamentals, rather than simply acting as corporate 
arbitrageurs. 

Such considerations suggest that the practical approach is to combine several 
imperfect measures. Candidate methods of measuring sentiment (ordered from 
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origins in investor psychology to responses by corporate insiders) include surveys; 
mood proxies; retail investor trades; mutual fund flows; trading volume; premia on 
dividend-paying stocks; closed-end fund discounts; option implied volatility; first- 
day returns on initial public offerings (IPOs); volume of initial public offerings; 
new equity issues; and insider trading. We comment on these sentiment proxies 
and then choose among them. 

Investor Surveys. Perhaps just by asking investors how optimistic they are, we can 
gain insight into the marginal irrational investor. Robert Shiller has conducted 
investor attitude surveys since 1989. UBS/Gallup surveys randomly-selected inves- 
tor households, and Investors Intelligence surveys financial newsletter writers; 
Brown and Cliff (2005) use the latter to forecast market returns. Qiu and Welch 
(2006) point out that although consumers polled for the University of Michigan 
Consumer Confidence Index are not asked directly for their views on securities 
prices, changes in that Consumer Confidence Index nonetheless correlate highly 
with changes in the UBS/Gallup index. They and Lemmon and Portniaguina 
(2006) show that changes in the consumer confidence measure correlate especially 
strongly with small stock returns and the returns of firms held disproportionately by 
retail investors. 

Investor Mood. Some papers have creatively tried to connect stock prices to 
exogenous changes in human emotions. Kamstra, Kramer, and Levi (2003) find 
that market returns are on average lower through the fall and winter, which they 
attribute to the onset of seasonal affective disorder, a depressive disorder associated 
with declining hours of daylight. They report patterns from different latitudes and 
both hemispheres which also appear consistent with this interpretation. Edmans, 
Garcia, and Norli (forthcoming) use international soccer results as a mood variable 
and find that losses in major games predict poor returns in the losing country the 
next day, particularly among small stocks. 

Retail Investor Trades. The inexperienced retail or individual investor is more 
likely than the professional to be subject to sentiment. Greenwood and Nagel 
(2006) find that younger investors were more likely than older investors to buy 
stocks at the peak of the Internet bubble. More generally, Barber, Odean, and Zhu 
(2006) and Kumar and Lee (forthcoming) find in micro-level trading data that 
retail investors buy and sell stocks in concert, which is consistent with systematic 
sentiment. Kumar and Lee suggest constructing sentiment measures for retail 
investors based on whether such investors are buying or selling. 

Mutual Fund Flows. Data are easily available on how mutual fund investors 
allocate across fund categories. Brown, Goetzmann, Hiraki, Shiraishi, and Wa- 
tanabe (2002) propose an overall market sentiment measure based on how fund 
investors are moving into and out of, for example, “safe” government bond funds 
and “risky” growth stock funds. Mutual fund investors are well-known to chase 
investments with high recent returns (for example, Warther, 1995), so whether the 
causality also goes the other direction—whether their allocation decisions actually 
lead to mispricing—is a tricky question. Frazzini and Lamont (2006) find some 
affirmative evidence by using fund flows to proxy for sentiment for individual 
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stocks. They find that when funds holding a particular stock experience strong 
inflows, the subsequent performance of that stock is relatively poor. 

Trading Volume. Trading volume, or more generally liquidity, can be viewed as 
an investor sentiment index. For instance, Baker and Stein (2004) note that if 
short-selling is costlier than opening and closing long positions (as it is, in prac- 
tice), irrational investors are more likely to trade, and thus add liquidity, when they 
are optimistic and betting on rising stocks rather than when they are pessimistic 
and betting on falling stocks. In Scheinkman and Xiong (2003), volume reveals 
underlying differences of opinion, which are in turn related to valuation levels 
when short selling is difficult. Market turnover, the ratio of trading volume to the 
number of shares listed on the New York Stock Exchange, is a simple proxy for this 
concept. 

Dividend Premium. Dividend-paying stocks resemble bonds in that their predict- 
able income stream represents a salient characteristic of safety. The first price-based 
measure we mention here is therefore the “premium” for dividend-paying stocks, 
which may be inversely related to sentiment. In Baker and Wurgler (2004a, b), we 
define the dividend premium as the difference between the average market-to- 
book-value ratios of dividend payers and nonpayers. The dividend premium ex- 
plains well the major historical trends in firms’ propensity to pay dividends, such as 
the post-1977 decline in this propensity documented by Fama and French (2001); 
that is, when dividends are at a premium, firms are more likely to pay them, and are 
less so when they are discounted. In other words, on the margin, firms appear to 
cater to prevailing sentiment for or against “safety” when deciding whether to pay 
dividends. 

Closed-End Fund Discount. Closed-end funds are investment companies who 
issue a fixed number of shares, which then trade on stock exchanges. The closed- 
end fund “discount” (or occasionally premium) is the difference between the net 
asset value of a fund’s actual security holdings and the fund’s market price. Many 
authors, including Zweig (1973), Lee, Shleifer, and Thaler (1991), and Neal and 
Wheatley (1998), have argued that if closed-end funds are disproportionately held 
by retail investors, the average discount on closed-end equity funds may be a 
sentiment index, with the discount increasing when retail investors are bearish. 

Option Implied Volatility. Options prices rise when the value of the underlying 
asset has greater expected volatility, and options pricing models such as the 
Black-Scholes formula can be inverted to yield implied volatility as a function of 
options prices. The Market Volatility Index (“VIX”), which measures the implied 
volatility of options on the Standard and Poor’s 100 stock index, is often called the 
“investor fear gauge” by practitioners. Whaley (2000) discusses the spikes in the VIX 
series since its 1986 inception, which include the crash of October 1987 and the 
1998 Long Term Capital Management crisis. 

IPO First-Day Returns. Initial public offerings sometimes earn such remarkable 
returns on their first trading day that it is difficult to find an explanation that does 
not involve investor enthusiasm. For example, Netscape’s return on the day of its 
August 1995 IPO was 108 percent. Interestingly, IPO first-day returns are not 
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idiosyncratic. Average first-day returns display peaks and troughs which are highly 
correlated with IPO volume (discussed next) and other sentiment proxies that are 
not fundamentally related.* 

IPO Volume. The underlying demand for initial public offerings is often said to 
be extremely sensitive to investor sentiment. Investment bankers speak of “windows 
of opportunity” for an initial public offering that capriciously open and close. Such 
caprice could explain why IPO volume displays wild fluctuations, with a rate of over 
100 issues per month in some periods and zero issues per month in others. 

Equity Issues Over Total New Issues. A broader measure of equity financing activity 
is the equity share of total equity and debt issues by all corporations. This measures all 
equity offerings, not just IPOs. In Baker and Wurgler (2000), we find that high values 
of the equity share portend low stock market returns, and suggest that this pattern 
reflects firms shifting successfully between equity and debt to reduce the overall cost of 
capital. This pattern need not imply that individual firms or their managers can predict 
prices on the market as a whole. Rather, correlated mispricings across firms may lead 
to correlated managerial actions, which may then forecast correlated corrections of 
mispricings—that is, forecast market returns. 

Insider Trading. Corporate executives have better information about the true 
value of their firms than outside investors. Thus, legalities aside, executives’ per- 
sonal portfolio decisions may also reveal their views about the mispricing of their 
firm. If sentiment leads to correlated mispricings across firms, insider trading 
patterns may contain a systematic sentiment component. See Seyhun (1998) for 
evidence on the ability of insider trading activity to predict stock returns. 


A Sentiment Index 

Which of the above measures to choose? Data availability narrows the list 
considerably. Sentiment may vary daily, but major episodes occur over years, and 
the most convincing tests of the effects of sentiment are those in which it is used to 
actually predict long-horizon returns—tests which suggest a contrarian trading 
strategy. This rules out using data that do not go back as far as our stock returns 
data (that is, to the 1960s), which would exclude, for example, data on insider 
trading; micro-level data on trading behavior; and implied volatility series. 

Instead, we construct an index based on the six proxies we use in Baker and 
Wurgler (2006): trading volume as measured by NYSE turnover; the dividend 
premium; the closed-end fund discount; the number and first-day returns on IPOs; 
and the equity share in new issues. All these data are available at <http://www.stern. 
nyu.edu/~jwurgler>. Later on, we will also consider some mutual fund series. 


* Why IPOs are “underpriced” to such an extreme degree is still a puzzle, since the prices are set in 
consultation with investment bankers well-informed about market conditions. The extreme unpredict- 
ability of investor sentiment may be a factor. The offer price is typically set in advance and indications 
of interest are then gathered from potential investors. It may be better to issue shares at a likely discount 
to first day prices than risk an undersubscribed offering in a period of generally high sentiment and 
valuations. This raises the question of why companies do not simply auction their shares. See Ritter (2003) 
for a discussion of underpricing and Ljungqvist, Nanda, and Singh (2006) for a sentiment-based model. 
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Although these six proxies are highly correlated in the expected directions, 
some of them also contain idiosyncratic components that are unrelated to senti- 
ment. For example, the 1975 deregulation of brokerage commissions and the 
subsequent long decline in trading costs has led to a decades-long upward trend in 
turnover, so we use the log of turnover minus a five-year moving average. With 
respect to closed-end fund discounts, if the majority of individual investors have 
come to prefer open-end funds in recent years, the discount provides a less useful 
summary of the opinion of the marginal investor than it once did. And the 
evolution of public debt markets has made the equity share less useful in recent 
years. But no obvious patch suggests itself in these cases. 

Also, some of the sentiment proxies reflect economic fundamentals to some 
extent. For instance, IPO volume depends, in part, on prevailing investment 
opportunities. To remove such influences, at least partially, we regress each proxy 
on a set of macroeconomic indicators—growth in industrial production, real 
growth in durable, nondurable, and services consumption, growth in employment, 
and an NBER recession indicator—and use the residuals from these regressions as 
our sentiment proxies. 

The six sentiment proxies will have a common sentiment component (espe- 
cially given that major macroeconomic influences have been removed), so we can 
iron out the remaining idiosyncracies by averaging them together into an index. 
We form a sentiment-levels index to test for return predictability conditional on the 
state of sentiment and also a sentiment-changes index to test for return co- 
movement patterns associated with changes in sentiment. The levels index is simply 
the first principal component of the six proxies.” The changes index is the first 
principal component of the changes in the six proxies.° 

Figure 2 shows the sentiment indexes graphically. As expected, variables 


° Brown and Cliff (2004) also use a principal components methodology to define a sentiment index. The 
first principal component of a set of time series variables is simply the linear combination of the variables 
with the coefficients chosen to capture as much of the joint variation across the series as possible. The 
second principal component performs the same analysis but defines the relevant series as the residuals 
from the first principal component—and so on. In defining which of our six series to include in the 
analysis, there is a decision to be made concerning timing. Some variables may reflect the same shift in 
sentiment sooner than others. In general, proxies that involve firm supply decisions are further down 
the chain of events and likely to lag behind proxies that are based directly on investor trading patterns 
or prices. In Baker and Wurgler (2006), we find that the best combination to capture the common 
variation in annual series includes the current values of the closed-end fund discount, the equity share, 
and IPO volume, and one-year-lagged versions of the three other variables. We are using monthly data 
here, but for simplicity, we also adopt this convention. 

ê While we could simply take the changes in the sentiment levels index, a better approach is to form a 
second index based on the first principal component of the changes in the six proxies. The reason for 
this preference is that the proxies have differential noisiness in going from levels to changes. For 
example, turnover has low frequency error related to falling transaction costs, but high frequency 
changes are more informative. The equity share, on the other hand, has low frequency error related to 
long-term shifts in financing patterns plus high frequency error because corporations can respond to 
sentiment only with a lag (which itself is unlikely to be consistent enough to try to align the changes in 
the series). Hence, equity share fluctuations will feature prominently in a changes-in-the-levels index, 
but will likely drop out in an index based on changes. 
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Figure 2 


A Sentiment Index, January 1966 through December 2005 
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Note: This figure shows the first principal component of levels and changes in six measures of sentiment: 
the closed-end fund discount (CEFD), detrended log turnover (TURN), number of IPOs (NIPO), 
first-day return on IPOs (RIPO), dividend premium (PDND), and equity share in new issues (S), each 
standardized and with the effect of macroeconomic conditions removed. In the levels index, turnover, 
the first-day return on IPOs, and the dividend premium are lagged 12 months. Both indices are 
standardized to have zero mean and unit variance over the 40 year period. 


positively associated with sentiment levels include share turnover (TURN), IPO 
volume (NIPO), IPO first-day returns (RIPO), and the equity share in new issues 
(S), and those negatively associated are the closed-end fund discount (CEFD) and 
the dividend premium (PDND). The bottom panel reports the changes index. The 
coefficients all have the same signs as in the levels index, with the exception of the 
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equity share. We regard its unexpected sign as a chance event made possible by the 
fact that its changes at high frequencies are largely unrelated to sentiment, but we 
retain it to avoid data mining. 

While data availability is the key constraint, some judgment has entered this 
approach to measuring sentiment. Robustness is a natural concern. We offer two 
remarks in response. First, the process of averaging the six proxies is not crucial. 
They are strongly correlated and if they were each studied as independent senti- 
ment indexes, some would display empirical results even stronger (that is, more 
consistent with a strong role for sentiment) than those we present below. We 
pursue the index approach so as not to elevate individual proxies arbitrarily and to 
iron out idiosyncratic variation. Second, in Baker and Wurgler (2006), we find that 
macro fundamentals explain little of the common variation in the six series. In 
other words, indexes formed from the “raw” series would look and perform almost 
identically to those used here. Nonetheless we include this step to illustrate an 
approach to controlling for fundamentals. 


Does This Index Capture Major Fluctuations In Sentiment? An Eyeball Test 

Perhaps the best evidence that the index generally succeeds in capturing 
sentiment is simply that it lines up fairly well with the anecdotal accounts of bubbles 
and crashes written by authors such as Brown (1991), Dreman (1979), Graham 
(1973), Malkiel (1999), Shiller (2000), and Siegel (1998). The first major bubble in 
our data period developed in 1967 and 1968. Brown (1991, p. 90) writes that 
“scores of franchisers, computer firms, and mobile home manufactures seemed to 
promise overnight wealth. . .. [while] quality was pretty much forgotten.” The early 
1970s, on the other hand, are invariably described in bear market terms. Yet a set 
of established, large, stable, consistently profitable stocks known as the “Nifty Fifty” 
enjoyed extreme price—earnings ratios. Siegel (1998, p. 106) writes, “All of these 
stocks had proven growth records, continual increases in dividends ... and high 
market capitalization.” The Nifty Fifty is a mirror image of the speculative episodes 
that occurred before and after it, which generally involved small, young, unprofit- 
able growth stocks in high-sentiment periods. 

The late 1970s through mid-1980s are described anecdotally as a period of 
generally high sentiment. This period witnessed a series of speculative episodes, 
including those involving gambling issues in 1977 and 1978; natural resource 
startups in 1980 on the heels of the second oil crisis (Ritter, 1984); and the 
high-tech and biotech booms in the first half of 1983. The latter two episodes are 
particularly evident in the sentiment levels index. But by 1987 and 1988, Malkiel 
(1999, p. 80) writes, “market sentiment had changed from an acceptance of an 
exciting story ... to a desire to stay closer to earth with low-multiple stocks that 
actually pay dividends.” Consistent with this view, the overall index shows sentiment 
at a high level during the early 1980s and tailing off somewhat toward the end of 
the decade. 

The late-1990s bubble in technology stocks will be familiar to many readers. By 
all accounts, sentiment was broadly high before the bubble started to burst in 2000. 
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Malkiel (1999) draws parallels to episodes in the 1960s, 1970s, and 1980s, and 
Shiller (2000) and others compare the Internet bubble to that of the late 1920s. 
The sentiment index flags 1999 as a high-sentiment year, and the dividend pre- 
mium and first-day returns on IPOs hit record levels that year. 

The sentiment changes index in the bottom of Figure 2 is harder to decipher 
in an eyeball test. However, when the series is viewed in light of major speculative 
episodes, one pattern does appear: The volatility of sentiment rises in a speculative 
episode. This pattern suggests that the relative influence of fundamentals and 
sentiment on aggregate market returns changes over time. 


Mutual Fund Flows 

Detailed data on mutual fund flows are not available back to the 1960s, so we 
do not include these in our main indexes. However, we use the period of overlap 
to correlate patterns in fund flows and the indexes. This exercise is useful because 
fund flows provide a transparent measure of decisions made by a large set of 
investors who are, on average, less sophisticated and more likely to display senti- 
ment. Moreover, the fund flows data help us to investigate the precise mechanism 
through which sentiment affects stock prices, as we explain. 

The Investment Company Institute offers monthly data on flows into various 
categories of funds. We look at net flows into the eight stock-oriented fund 
categories for which data exist back to January 1990. The categories vary from 
speculative “aggressive growth” funds to safer, dividend-paying “income” funds. 

Figure 3 shows the results of a principal components analysis of changes in 
fund flows, as in Goetzmann, Massa, and Rouwenhorst (2000). Once more, the 
principal components analysis helps us detect general patterns across several time 
series while ironing out distracting idiosyncratic fluctuations. The results in the 
figure show that across the eight stock fund categories, the first principal compo- 
nent is a “general demand” effect. It reflects the fact that investors often shift in and 
out of stock funds en masse. In contrast, the second principal component shows 
that the next most important effect is shifts between more speculative funds and 
safer funds. Thus we call this the “speculative demand” component. Therefore, 
controlling for the overall generic equity fund demand, when flows fall in the more 
speculative categories they tend to rise in the less speculative categories. Barberis 
and Shleifer (2003) call this “style investing.” 

With these two principal components in hand, we can construct monthly time 
series of the two most important sources of changes in mutual fund flows: one 
reflecting general demand and another reflecting speculative demand. We then 
correlate these with the sentiment changes index. During the period of overlap, the 
sentiment changes index has a marginally significant correlation of 0.16 with 
general fund demand and, perhaps more interestingly, a highly significant corre- 
lation of 0.36 with speculative demand. This latter correlation is particularly sug- 
gestive that our overall sentiment indexes do, to a large extent, capture a prevailing 
“greed” versus “fear” or “bullish” versus “bearish” notion. 

Finally, recall that there are essentially two distinct channels by which senti- 
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Figure 3 
Principal Components of Equity Mutual Fund Flow Changes, January 1990 through 
December 2005 


Panel A: Generic demand: Coefficients on the first principal component of flow 
changes 
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Note: Panels A and B show the contribution of a one-standard-deviation change in each equity-oriented 
mutual fund category to the first two principal components of changes in mutual fund flows, which we 
label general and speculative demand components. Each bar shows the impact of a one-standard- 
deviation change in mutual fund flow. For example, a one-standard-deviation change in flows into 
aggressive growth funds increases the first principal component by 0.23 and the second principal 
component by 0.29. Flows are net sales minus redemptions by category, scaled by total mutual fund 
assets across categories. 
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ment will have cross-sectional effects: when general investor demand for risky assets 
is uniform but stocks differ in the costs and risks of arbitrage, and when investor 
demand focuses on relatively speculative stocks and the difficulty of arbitrage is 
held constant. The fairly clean empirical break of fund flows into general and 
speculative demand components suggests the possibility of empirically separating 
the two channels. Of course, both can operate at the same time. 


Using Sentiment to Explain Current Returns 


With the index of sentiment changes and the fund flow series in hand, we 
now turn to testing the key hypotheses about how sentiment affects stocks. We start 
by asking whether speculative and harder-to-arbitrage stocks are relatively more 
affected by sentiment changes. At the end of the section, we briefly consider effects 
on the aggregate market. 


Defining Speculative, Difficult-to-Arbitrage Stocks 

To study the differential effects of investor sentiment across firms, we first need 
a way of sorting stocks according to their speculative appeal and their difficulty of 
arbitrage. A natural proxy for speculative appeal would be the dispersion of 
professional analysts’ earnings forecasts for that company, but such forecasts are 
not available for all stocks back to the mid-1960s. A possible proxy for difficulty of 
arbitrage could be a direct measure of transaction costs for given stocks, but those 
too are unavailable for a long time series. 

We sort stocks according to their recent return volatility, specifically the 
standard deviation of monthly returns over the prior year. Returns data are from 
the Center for Research in Securities Prices (CRSP). High volatility is characteristic 
of stocks with strong speculative appeal; low volatility is a bond-like feature. More- 
over, highly volatile stocks are generally riskier to arbitrage, so an arbitrageur with 
limited risk-bearing capacity will hesitate before making large bets against mispric- 
ing. Each month, we place each stock into one of ten portfolios according to the 
decile of their return volatility of the previous year, and we use the returns on the 


resulting portfolios to represent the cross-section of stock returns.’ 


Sentiment Betas: Cross-Sectional Effects of Sentiment Changes 
Figure 4 shows the relationship between sentiment changes and the returns on 
the ten volatility portfolios. The dependent variable is the monthly return on one 


7In Baker and Wurgler (2006), we divide stocks in several other ways, including firm age, market 
capitalization, dividend payment, and profitability. These are natural given an intuition that larger, 
older, dividend paying, and profitable firms tend to be more bond-like and generally easier to arbitrage. 
We also sort stocks on growth and distress indicators such as the market-to-book equity ratio, asset 
growth, and sales growth. When sorted this way, bond-like stocks now lie in the middle deciles while 
speculative stocks, whether due to their extreme growth potential or risk of financial distress, are found 
at both extremes. These other sorting methods produce qualitatively similar results, so we report results 
only for volatility portfolios here. 
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Figure 4 


The Sensitivity of Returns of Different Types of Stock to Investor Sentiment 


Panel A. Sentiment betas based on mutual fund flows 
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Panel B. Sentiment betas based on a sentiment index 
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Note: The monthly returns of volatility sorted portfolios (more volatility represents more speculative, 
difficult-to-arbitrage stocks) are regressed on general and speculative demand components of mutual 
fund flow changes (Panel A) and the index of sentimen changes (Panel B). The coefficients, or 
sentiment betas, show the effect of a one-standard-deviation difference in the sentiment measure on 
average returns in percentage points. The regressions control for the value-weighted stock market 


return. 


of the ten volatility portfolios. For each one of these ten portfolios, we run three 
time-series regressions. In Panel A, we plot the coefficients on the general or 
speculative mutual fund demand factors. In Panel B, we plot the coefficients on the 
sentiment changes index. Each of the regressions also includes the value-weighted 
market return as a control variable, because high volatility stocks are likely to have 


146 Journal of Economic Perspectives 


higher market betas, an effect that we do not want to contaminate the sentiment 
betas. 

The results are as predicted. The effect of general demand for stock funds on 
monthly returns is higher for higher volatility portfolios, presumably because stocks 
therein are harder to arbitrage. In addition, the effect of speculative demand is also 
increasing, presumably reflecting the more speculative nature of volatile stocks. 
The convex pattern in the coefficients is intriguing—stocks in the extreme volatility 
decile, which are often small, rapidly growing, or in financial distress, are dispro- 
portionately more sensitive to both components of fund flows. 

Sentiment betas also increase as stocks become more speculative and harder to 
arbitrage.® Figure 4, Panel B, shows that controlling for market returns, a one- 
standard-deviation increase in the sentiment changes index increases returns on 
the eighth volatility decile portfolio, for example, by about one percentage point. 
The effect on the tenth decile portfolio is over two percentage points. For partic- 
ularly bond-like stocks, on the other hand, the effect is slightly negative. This is 
consistent with the most bond-like stocks actually having slightly negative sentiment 
betas, as conjectured in Figure 1. Once again, there is a convex pattern in the 
sentiment betas. 

Although these results are all highly consistent with the seesaw diagram, there 
are other interpretations. For example, perhaps speculative flows are highly corre- 
lated with the returns of speculative stocks simply because mutual fund investors 
chase returns, not because their trading has any causal effect on its own. Or, the 
sentiment changes index may include components, such as the dividend premium, 
which lead to mechanical differences in the correlations between sentiment 
changes and different stocks (for example, dividend-paying and -nonpaying 
stocks) .° Or, the sentiment index may, despite our best efforts, be contaminated by 
economic fundamentals, which should, of course, affect returns independently. 
The fact that sentiment actually helps to predict returns, as we illustrate below, 
suggests that these considerations cannot fully account for the patterns in Figure 4. 


Aggregate Effects 

Although our main focus is on cross-sectional differences, a positive correla- 
tion will exist between aggregate market returns and sentiment changes if the 
average stock is affected by sentiment. Indeed, the correlation between an 
equal-weighted market return index and the sentiment changes index is a highly 
significant 0.43. The correlation between equal-weighted returns and speculative 
demand estimated from fund flows is 0.26, and the correlation between equal- 
weighted returns and general demand from fund flows is 0.48. Using capitalization- 


8 See Glushkov (2006) for sentiment betas for portfolios sorted on other characteristics as well (as 
opposed to only volatility). All of his results are closely consistent with the argument that hard-to- 
arbitrage and value stocks are more sensitive to sentiment. 

° Excluding any particular component from the sentiment indexes, including the dividend premium, 
does not alter our conclusions. 
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weighted market returns, which give more weight to large firms, the respective 
correlations are 0.32, 0.29, and 0.39, again all highly significant. 


Using Sentiment to Predict Stock Returns 


The strongest tests of the effects of sentiment involve return predictability. If 
high sentiment indeed causes overvaluation, we may be able to document low 
future returns on sentiment-prone stocks as sentiment wanes and fundamentals are 
revealed. Predictability is not a natural implication of the skeptical view that the 
correlation between returns and sentiment indices arises because the latter are 
contaminated by fundamentals, for example. 


Cross-sectional Predictability 

To test these ideas further, we create an empirical version of the sentiment 
seesaw and compare it to the predictions in Figure 1. We use the same volatility- 
based characterization of stocks to identify those that are speculative and difficult 
to arbitrage. Next, we split the time series into high- and low-sentiment periods 
using the previous month’s measure of the sentiment level. Finally, we compute 
average returns for each of the ten volatility portfolios, for the two separate periods 
and overall. As with the calculation of sentiment betas, we control for the value- 
weighted market return. 

The resulting picture, in Figure 5, is strikingly similar to the predictions of the 
seesaw diagram. When sentiment is low, the average future returns of speculative 
stocks exceed those of bond-like stocks. When sentiment is high, the average future 
returns of speculative stocks are on average lower than the returns of bond-like 
stocks. This pattern is a telling one—the fact that riskier stocks (at least, stocks that 
are riskier by all outward appearances) sometimes have lower expected returns is 
inconsistent with classical asset pricing in which investors bear risk because they are 
compensated by higher expected return. 

The unconditional average returns are slightly lower for speculative stocks, 
consistent with behavioral models of disagreement among investors combined with 
short-sales constraints (such as Hong and Stein (2003) and other models they 
discuss in the current issue of this journal). The market-adjusted returns are on 
average positive because of the well-known size effect—in January, small capitali- 
zation stocks earn high returns, on average—which increases the average return of 
our equally-weighted portfolios. Controlling for equal-weighted market returns 
instead of value-weighted returns shifts the market-adjusted returns down across all 
ten portfolios, but the overall similarity to Figure | remains intact. 


Aggregate Predictability 

When sentiment is high, subsequent market returns are low. Figure 6 shows 
that when the sentiment level is more than one standard deviation above its 
historical average, monthly returns average —0.41 percentage points for equal- 
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Figure 5 
Sentiment and Future Returns 
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Note: This figure shows average monthly returns over the value-weighted market index for volatility 
sorted portfolios following a month with a positive sentiment level (short dash) and with a negative 
sentiment level (long dash), as well as the overall average market-adjusted return (solid line). The latter 
are positive because of the small firm effect over this period. The volatility portfolios are equal weighted 
while the market index is value weighted. 


weighted market index returns and —0.34 percentage points for value-weighted 
returns. And when the investor sentiment level is very low, for example, more than 
one standard deviation below its historical average, monthly returns average 2.75 
and 1.18 percentage points for equal- and value-weighted indexes, respectively.'° 
Therefore, just as the correlation between sentiment changes and returns is higher 
for an equal-weighted index of returns, so is the correlation between sentiment 
levels and subsequent equal-weighted stock returns. The gap between equal- 
weighted and value-weighted market returns again demonstrates that the impact of 
sentiment is stronger on small stocks, as predicted. 

While Figure 6 indicates economically important gaps, the statistical signifi- 
cance is modest, as is the case with other arguably nonsentiment predictors of 
aggregate returns such as the dividend-price ratio. Put another way, market crashes 
tend to occur in high-sentiment periods, but the timing of the crashes within these 
periods is very hard to predict. 


10 For Figure 6, we break the historical time series into four sentiment states, while for Figure 5, we break 
it into only two. We made the latter choice to allow for an easier comparison to the seesaw diagram. If 
Figure 5 were to display results for four sentiment states, it would display all of the expected patterns— 
for example, when sentiment is very high, the highest-volatility stocks subsequently earn particularly low 
returns, and so on. 
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Figure 6 
Sentiment and Market Returns 
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Note: Average monthly returns in percentage points on the equal- and value-weighted market portfolios. 
The sample is divided into four groups according to the sentiment level in the preceding month. SD is 
standard deviation. 


Conclusion 


This paper outlines a “top down” approach to behavioral finance and the stock 
market. We take the origin of investor sentiment as exogenous and instead focus on 
its empirical effects. We show that it is quite possible to measure investor sentiment, 
and that waves of sentiment have clearly discernible, important, and regular effects 
on individual firms and on the stock market as a whole. In particular, stocks that are 
difficult to arbitrage or to value are most affected by sentiment. The sentiment 
seesaw diagram in Figure 1 summarizes our approach. 

Looking forward, the investor sentiment approach faces a number of chal- 
lenges: characterizing and measuring uninformed demand or investor sentiment; 
understanding the foundations and variation in investor sentiment over time; and 
determining which particular stocks attract speculators or have limited arbitrage 
potential. Much remains to be done in terms of spelling out this framework, but the 
potential payoffs of an improved understanding of investor sentiment are substan- 
tial. For example, the standard methodology for estimating fundamental market 
betas (an input to long-term capital budgeting and other important financial deci- 
sions) does not account for sentiment. Doing so might improve estimates and clarify 
their interpretation; Shefrin (2005) considers this issue. Also, we have seen that 
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sentiment affects the cost of capital. Therefore it may have real consequences for the 


allocation of corporate investment capital between safer and more speculative firms. 
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